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(57) ABSTRACT 

A computer system and method of operation thereof are 
provided that allow interactive navigation and exploration of 
spatial environments, both real and virtual. The computer 
system employs a data architecture comprising a network of 
nodes connected by branches. Each node in the network 
represents an intersection in the real environment that allows 
a user of the computer system to select which path to follow. 
Likewise, each branch in the network represents a path 
connecting physical intersections in the real environment. 
The network is constructed directly from a map of the target 
environment. Navigation data such as image frame 
sequences, intersections between paths, and other related 
information are associated with the elements of the network. 
This establishes a direct relationship between locations in 
the environment and the data which represent them. From 
such an organization, the user may tour the environment, 
viewing the image sequences associated with each path and 
choosing among intersecting paths at will. In addition to 
navigating through the environment, the user may also 
access auxiliary information that is related to particular 
points of interest. 

33 Claims, 22 Drawing Sheets 
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GRAPH-BASED VISUAL NAVIGATION 
THROUGH SPATIAL ENVIRONMENTS 

CROSS REFERENCE TO APPENDICES 

Microfiche Appendix A, which is part of the present 
disclosure, is an appendix consisting of two sheets of 
microfiche containing a total of 132 frames, microfiche 
Appendix B, which is part of the present disclosure, is an 
appendix consisting of single sheet containing a total of 44 
frames and microfiche Appendix C f which is part of the 
present disclosure, consists of a single sheet containing 34 
frames of a computer program and related data, according to 
an embodiment of the invention. These listings of computer 
program contains material which is subject to copyright 
protection. The copyright owner has no objection to the 
facsimile reproduction by anyone of the patent document or 
the present disclosure, as it appears in the Patent and 
Trademark Office patent files or records, but otherwise 
reserves all copyright rights whatsoever. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates generally to computer pro- 
grams and, particularly, to computer systems used to navi- 
gate virtual environments. 

2. Related Art 

The most common method of presenting a virtual tour of 
a real, physical environment is film or videotape. While such 
a presentation offers high visual accuracy, the progress of the 
tour is generally limited to a single, static route chosen by 
the photographer or editor. In other words, the observer of 
the tour has none of the choices usually available to someone 
who is physically in the environment: choices such as 
turning one way or the other at an intersection, skipping 
uninteresting paths, or proceeding directly to a favorite 
location. The observer's interaction with the environment is 
limited to commands such as start, stop, fast forward or 
reverse. A further limitation of this approach is that the 
observer generally has no other context than the image, 
itself. That is, the limited field of view makes it difficult for 
the observer to orient him/herself in a complex environment. 

For tours of virtual environments, computer systems are 
commonly used to render images from models representing 
the virtual environments. As opposed to videotaped 
presentations, such systems can be highly interactive since 
the images are rendered in real time. There is, however, a 
tradeoff between cost and visual accuracy. Computer sys- 
tems capable of rendering very high-resolution images in 
real time are prohibitively expensive, while more affordable 
systems produce inferior quality images, 

SUMMARY OF THE INVENTION 

The present invention relates to a computer system which 
allows interactive navigation and exploration of spatial 
environments, both real and virtual. The computer system 
employs a data architecture comprising a network of nodes 
connected by branches. Each node in the network represents 
an intersection in the real environment that allows a user of 
the computer system to select which path to follow. 
Likewise, each branch in the network represents a path 
connecting physical intersections in the real environment. 

The network is constructed directly from a map of the 
target environment. Navigation data such as image frame 
sequences, intersections between paths, and other related 
information are associated with the elements of the network. 
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This establishes a direct relationship between locations in 
the environment and the data which represent them. From 
such an organization, the user may tour the environment, 
viewing the image sequences associated with each path and 

5 choosing among intersecting paths at will. 

In addition to navigating through the environment, the 
user may also access auxiliary information that is related to 
particular points of interest. This auxiliary information can 
be of varying forms: video, audio, still images, etc. Such 

30 information is linked to a point of interest through the 
associated network element. By associating this information 
with the virtual environment, the computer system of the 
present invention is able to provide the observer with an 
enhanced view of the real environment represented by the 

35 virtual environment. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram that illustrates the architecture 
2Q of the principal design constructs of the present invention; 

FIG. 2 illustrates an example of the network of nodes and 
branches that is the underlying representation of the navi- 
gation model; 

FIG. 3 illustrates one possible map that can be realized 
25 from the network shown in FIG. 2; 

FIG. 4 is a functional flow diagram that illustrates the 
logic and actions of the image frame sequence processing 
function; 

FIGS. 5a, 5b, and 5c illustrates the rotation of path vectors 
30 from absolute to user-relative coordinates; 

FIG. 6 is a functional flow diagram that illustrates the 
process of initializing a new branch for playback. 

FIG 7 illustrates the calculation of branches and image 
35 frames for the Browse event. 

FIG 8 illustrate the hardware and software structure of a 
computer system, in accordance to an embodiment of the 
invention. 

FIGS. 9A and 9B illustrates the derivation of the software 
40 classes that comprise the application library. 

FIG, 10 illustrates the derivation of authoring classes 
from the application library. 

FIGS. 11-16 show a window displayed on a screen of a 
45 display device of the computer system of FIG. 8, at succes- 
sive points during navigation of a virtual environment. 

FIGS. 17-22 show a window displayed on a screen of a 
display device of the computer system of FIG. 8, at succes- 
sive points during authoring of a virtual environment. 

50 DETAILED DESCRIPTION OF THE 

INVENTION 

As disclosed herein, the present invention includes a 
method and a computer software system for composing and 

55 executing interactive tours of spatial environments. FIG. 8 
illustrates the hardware and software context of the present 
invention. The computer hardware specified is representa- 
tive of modern multimedia computer systems. Such systems 
contain a central processing unit (CPU) and memory, a video 

60 card to generate images, and a sound card to generate sound. 
The images are displayed on an electronic computer display 
monitor. One example of a suitable CPU and memory for the 
computer system is a Pentium processor, 200 Mhz, with 32 
MB RAM. The display must be capable of 16-bit color. 

65 The computer hardware further includes a keyboard and 
pointing device for data input. As is understood by those 
skilled in the art, the pointing device may be a mouse, 
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trackball, joystick, or other similar device. The computer 
system further includes a mass storage device and/or a 
network interface. Either a mass storage device or a com- 
puter network may be used to store large data such as image 
files. Suitable mass storage devices will have the character- 
istics of very large capacity as well as very fast, random 
access. An example of a suitable mass storage device is a 
CD-ROM drive. An example of a suitable computer network 
is the World Wide Web. 

The software context described in FIG. 8 includes a 
computer operating system and several service layers of 
computer software. Examples of suitable operating systems 
are Windows95, Windows98 and Windows NT 4.0, avail- 
able from Microsoft Corp. of Redmond, Wash. The present 
invention requires services provided by three software ser- 
vice applications, NetShow, DirectShow, and DirectX/ 
Direct3D, also available from Microsoft Corp. As illustrated 
in FIG. 8, the application software of the present invention 
has interfaces with each of these three service applications. 

Those skilled in the art realize that other hardware/ 
software platforms can be used in lieu of the ones described 
herein in accordance to the principles of the present inven- 
tion. Accordingly, the invention is not limited to any par- 
ticular hardware/software platform. 

FIG. 1 illustrates the architecture of the present invention. 
Rectangles in the figure identify design constructs, while 
arrows between rectangles identify relationships between 
constructs. As illustrated by the legend in the lower right end 
corner of FIG. 1, arrows with a single head on each end (e.g., 
arrow 121) indicate a one to one relationship between 
corresponding constructs, while arrows with a single head 
on one hand and a double head on the other end (e.g., arrow 
123) indicate a one-to-many relationship between corre- 
sponding constructs. 

Documents 101 are the primary construct of the 
architecture, representing a complete navigational model. 
Views 102 display some aspect of the document in a window 
of the computer display. Each view 102 is typically associ- 
ated with a display window. Relationship 123 indicates that 
a single document 101 may be displayed using multiple 
views 102. Relationship 125 indicates that three distinct 
types of views are used: main views 105, map views 106, 
and popup views 107. In one embodiment, main view 105 
displays image frame sequences associated with each path 
on the tour, as illustrated by arrow 128. Main view 105 also 
contains user controls 109, as illustrated by arrow 129. Map 
view 106, in turn, displays an interactive map of the 
environment, including tour paths, intersections, and points 
of interest (HotSpots) that allow users of the computer 
system to access auxiliary information. Popup views 107 
display auxiliary data 110 associated with HotSpots, as 
illustrated by arrow 130. 

Each document 101 may manage multiple streams 103, as 
illustrated by arrow 122. Each stream 103, in turn, provides 
data to a view 102, as illustrated by arrow 121. These data 
is contained in one or more stream files, whose names are 
stored in model database 104. Each document 101 is asso- 
ciated with a single model database 104 that contains the 
complete data for the document's navigation model, as 
illustrated by arrow 124. Model database 104 contains 
multiple model elements 108 that make up the navigation 
model, as illustrated by arrow 126. Relationship 132 indi- 
cates that four kinds of model elements are used. Nodes 118 
represent intersections. Pairs of nodes 118 are connected by 
branches 119, that represent paths between intersections, as 
illustrated by arrow 135. A single node 118 may serve as a 
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connecting point for two or more branches 119. HotSpots 
120 represent points of interest that enable users to access 
auxiliary data 110. HotSpots are placed along branches 119, 
as illustrated by arrow 137. Branch links 117 allow users to 

5 transition from one model document to another, as illus- 
trated by arrow 131. Branch links 117 allow traversal 
between independent navigation models, represented by 
different documents. Each model element 108 is displayed in 
a map view 106 as an element of the interactive map, as 

3Q illustrated by arrow 127. 

Relationship 133 indicates that auxiliary data 110 may 
take several forms. Model links 111 are similar to branch 
links 117 insofar as they connect documents 101, as illus- 
trated by arrow 139. However, while a branch link jumps 
5 unconditionally to a new model, replacing the current 
model, a model link initiates a new model within the context 
of the current model. When a new model is terminated, 
program execution returns to the location of the model link. 
A web link 112 connects the model to a World Wide Web 

2Q page. When a web link 112 is executed, the program 
launches a user's default web browser to access a URL 
associated with web link 112. In such case, the browser acts 
as a popup view 107. Auxiliary data 110 displayed in a 
popup view 107 may also include single images 113, image 

25 frame sequences 114, audio clips 115, and text documents 
116. When display of auxiliary data 110 is terminated, 
execution returns to the location of HotSpot 120 in the 
original model. Users may terminate display of auxiliary 
data 110 either by closing popup view 107 or by selecting 

30 one of the controls in main view 105. 

FIG. 2 illustrates a graphical representation of the navi- 
gational model, hereafter referred to as "the model". The 
model is conceptually a network of nodes and connecting 
branches. The network is represented as a directed, cyclic 

35 graph. Each node of the graph represents an endpoint or 
intersection in the model, and each branch represents a path 
in the model. For example, branch 207 connecting nodes 
202 and 203, represents a path in the model between the 
intersections represented by nodes 202 and 203. 

40 Furthermore, the orientation of the arrow on branch 207 
indicates that the associated path is directional, leading from 
node 202 to node 203. 

In one embodiment, the model of FIG. 2 is displayed to 
the user as a map of the environment, as shown in FIG. 3. 

45 Thus, intersections 301-306 of FIG. 3 correspond to nodes 
201-206 of FIG. 2. Paths on the map are not necessarily 
straight lines (i.e., a path may be composed of multiple, 
connected line segments). Path 307, constructed of two 
segments, corresponds to branch 207. In addition to inter- 

50 sections and paths, the map also displays HotSpots located 
along paths. For example, HotSpot 309 is located along path 
308. Since in the model HotSpots are associated with 
branches, HotSpot 309 is associated with branch 208 that, in 
turn, corresponds to path 308. 

55 Each branch in the model is associated with an image 
sequence that is displayed to the user during the course of 
the tour. In one embodiment, the sequence is a video clip 
shot while moving along the associated path in the real 
environment. The sequence may take other forms as well, 

60 such as computer-animated image frames showing a path 
through a virtual environment. Image sequences may reside 
on the host computer, or may reside on remote servers 
located anywhere on the World Wide Web. For instance, in 
some embodiments, image sequences may be shown using 

65 streaming video. 

HotSpots represent a linkage between an area of a navi- 
gation frame, typically an object in the frame, and a set of 
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associated data. A HotSpot is represented on the map by an Display Initial Image 

icon and in the main view image display by a bounded The initial stream is opened and the image frame associ- 

region of interest (ROI). A HotSpot' s associated data may ated with the starting node is displayed in main view 

take a variety of forms including, but not limited to, video window 1110. User controls are attached to main view 

sequences, animations, audio clips, text, still images, and 5 window 1110 and the stream is set to Pause, waiting for user 

World Wide Web links. When a HotSpot is activated during input, 

the course of the tour, the associated data is displayed in a Authoring 

popup view window. The general process of authoring is to construct a model 

Given the underlying structure of the data, the following by drawing elements in a map view window 1720 (FIG. 17) 

sections describe the operations that manipulate the data. 10 that are viewed in main view window 1710. Construction of 

The present invention provides three classes of capabilities: a model typically begins with a diagram of the environment, 

initialization, authoring (constructing and modifying the a set of image sequences, and auxiliary data. The diagram is 

model), and playback (navigating the model). displayed as a bitmap in map view window 1720. The user 

Overview of Program Execution defines model elements by drawing them over the diagram. 

When the program is started, a map view window 1120 15 For each path defined, the user selects an image sequence 

(FIG. 11) is initialized with intersections, paths, and from main view window 1710 and attaches the image 

HotSpots retrieved from model database 104. The default sequence to the path. For each HotSpot defined, the user 

starting intersection, as defined in model database 104, is the selects auxiliary data and associates the auxiliary data with 

initial start position. Main view window 1110 shows the the HotSpot. Editing operations allow the user to modify the 

image frame associated with the starting intersection, as well 20 contents of any model element. The authoring program 

as any path choices 1130 leading away from the starting enables the user to perform the following operations: 

intersection. Add Node: The user designates the location of the node on 

When the user selects a choice, the path represented by the map displayed in map view window 1720. 

that choice and any destination intersections are highlighted This operation draws an intersection icon at the desig- 

in map view window 1120 (e.g., path 1210 and node 1220 25 nated location, as shown in FIG. 17, creates the node in the 

in FIG. 12). The program plays the image sequence associ- model, and records the location of the intersection in the 

ated with the path (FIG. 12). Any HotSpots on the current node. 

path are highlighted on the map as the image sequence nears Delete Node: The user selects an intersection icon on the 

their locations, and the user receives both a visual and audio map displayed in map view 1720. This operation removes 

queue to the HotSpot's proximity (FIGS. 13-16). 30 the intersection icon from the map and deletes the associated 

As each image frame is decoded, the program updates the node from the model. If the node is the start or end point of 

current frame number (or sample time). When the current any branches, the branches are deleted as well (see operation 

frame approaches the end of the sequence range associated Delete Branch). 

with the current path, the program displays the path choices Edit Node: The user selects an intersection icon on the 

available at the upcoming intersection (FIG. 11). If the user 35 map displayed in map view window 1720. This operation 

does not select a choice before the image sequence reaches then displays a dialog 1910 (FIG. 19) wherein the user can 

the intersection, image playback is paused. If the user does modify any of the data contained in the associated node 

select a choice, image playback proceeds directly with the 1920. 

image sequence for the selected path (i.e. image playback is Add Branch: The user draws a path on the map displayed 

not paused at the intersection). 40 in map view window 1720. To draw the path, the user starts 

The user may also control program execution in other at any intersection and then draws a series of connected line 

ways. For example., the user can pause and continue image segments 1810 that end at another intersection (FIG. 18). 

playback at will, and can jump to an arbitrary intersection by The shape of the path should closely approximate the shape 

simply selecting a corresponding icon in map view window of the actual path followed to produce the image sequence 

1120. Similarly, the user can jump to anyHotSpot in the 45 for the path. The user then designates a start and an end of 

model by selecting a corresponding icon in map view the image sequence for the branch by selecting image frames 

window 1120. in main window 1710. This operation creates a branch in the 

Initialization model, records a specification of the path (including all 

In the case of the authoring tool, the program is initialized segments) in the branch, records references to the start and 

by simply starting the software application. In the case of the 50 end nodes in the branch, and adds a reference to the branch 

player the following operations are performed: to its start and end nodes. 

Startup Delete Branch: The user selects a path on the map 

This operation first creates a document, a model database, displayed in map view window 1720. This operation 

a stream object and a main view. A reference to the model removes the path from the map displayed in map view 

database is then passed to the stream object and a reference 55 window 1720, deletes the associated branch from the model, 

to the stream object is added to the document. and removes references to the branch from the two nodes 

Load Model File connected by the branch. 

This operation opens a model database file, connects the Edit Branch: The user first selects a path on the map 

main view to the stream object created during Startup, displayed in map view window 1720. Then a dialog 2010 is 

creates a map view, and connects the map view to the main 60 displayed to allow the user to modify any of the data 

view stream object. The model database created during contained in the associated branch 2020, as shown in FIG. 

Startup is then populated with the contents of the model 20. 

database file. A map is drawn using a bitmap file. Path, Add HotSpot: The user defines a HotSpot in main window 

intersections, and HotSpots from the model are overlaid on 1710 with the image sequence paused at the desired frame 

the background. The start position is initially set to the start 65 (the image sequence must already have been assigned to a 

node and the intersection icon is highlighted on a map branch). The user then defines a HotSpot area by drawing a 

displayed in map view window 1120, bounding figure around a region of interest 2120 in main 
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view window 1710 (FIG. 21). The user specifies data to be FIG. 4 is a flowchart of image frame processing operation 

associated with the HotSpot in a popup dialog 2110 (FIG. 400. Operation 400 first looks for certain conditions that 

21) . This operation uses the offset of the image frame within require special processing. When all conditions have been 
the sequence to interpolate a position along the associated handled, the next available image frame is rendered. First, 
path on the map. Based on this information a Hotspot icon 5 stage 405 determines whether the active stream is set to a 
2220 is drawn at a corresponding location on the map (FIG. Run state. If the stream state is not set to Run, image 

22) . playback is paused and operation 400 terminates. Next, 
Delete HotSpot: The user selects a HotSpot icon on the stage 410 determines whether the current frame is close to 

map displayed in map view window 1720. The HotSpot icon a HotSpot. Proximity is determined either with respect to a 

is removed from the map, the associated HotSpot is deleted 10 number of frames or to a time delta. If the test is true, in 

from the model and removes any references to the HotSpot stage 415 the HotSpot is activated (e.g., by highlighting the 

are deleted from all branches associated with the HotSpot. HotSpot icon in map view window 1120 and alerting the 

Edit HotSpot: The user selects a HotSpot icon on the map user to the proximity of the HotSpot via a visual and/or 

displayed in map view window 1720. A dialog box is then auditory queue). 

displayed to allow the user to modify any of the data 15 Stage 420 men determines whether the current frame is at 

contained on the corresponding HotSpot. the end of the current branch. Proximity is again determined 

Add Branch Link: The user draws a path on the map either with respect to a number of frames or to a time delta, 

displayed in map view window 1720, and optionally asso- If the test is true, path choices are activated in stage 425. 

ciates the path with an image sequence, in the manner Path choices are the set of paths branching out from the 

specified with respect to the Add Branch operation. The user 20 upcoming intersection. Activation includes first calculating 

then specifies both a model document in database 104 and a the set of path choices and then displaying the choices for 

node within that documents model as the endpoint of the user selection. Calculation is required because path choices 

branch link. This operation creates a branch link in the are always displayed relative to the current direction of 

model, records a specification of the path (including all travel (i.e. from the user's viewpoint), rather than according 

segments) in the branch link, records references to start and 25 to a fixed coordinate systems. FIG. Sa illustrates a sample 

end nodes in the branch link, and adds a reference to the scenario where path 501 enters intersection 502, and paths 

branch link to the start and end nodes. Note that the end node 503 and 504 leave intersection 502. If the current playback 

is not reached by traversing the branch link; the destination location is along path 501, the choices presented to the user 

is the remote node. at intersection 502 are to take either left path 503 or right 

Delete Branch Link: The user selects a path on the map 30 path 504. FIG. Sb illustrates these choices from the user's 

displayed in map view window 1720. The path is removed viewpoint. Since display of path choices is from the user's 

from the map, the associated branch link is removed from viewpoint, the user's current direction of travel corresponds 

the model, and only references to the branch link are to straight upscreen, 90 degrees left and right correspond to 

removed from the nodes connected by the branch link. straight left and right, respectively, on the screen, while 180 

Edit Branch Link: The user selects a path on the map 35 degrees backwards corresponds to straight downscreen. 

displayed in the map view 1720. A dialog box is then Calculation of path choices is accomplished as follows, 

displayed to allow the user to modify any of the data Recalling that a branch may represent a path of multiple line 

contained in the associated branch link. segments, the geometrical orientation of an outgoing branch 

Save Model: The user is prompted for a file name, and the is determined by a vector of a first line segment of the 

contents of the Model Database are written to the file 40 branch. The orientation of an incoming branch is likewise 

specified by the user determined by a vector of a last segment in the branch. The 

Playback incoming direction vector (representing the user's point of 

The playback capability allows a user to execute the view) is rotated such that the vector points "up" (i.e., the 

model, resulting in an interactive tour of the virtual envi- vector aligns with a 90 degrees angle). The difference angle 

ronment. There are two separate functions involved in 45 between the original orientation of the incoming vector and 

playback: processing image frames and handling user input. the rotated vector is the angle by which all of the outgoing 

Conceptually, the processing of image frames occurs itera- branch's vectors are rotated. In FIG. 5c, segment BA is the 

tively in the main processing loop of the program, while user vector of the incoming branch (path 501), and segments AC 

input is handled asynchronously. and AD are the vectors of outgoing branches 503 and 504. 

The program initiates the playback capability by prepar- 50 For each outgoing directional vector (xy), 
ing map view window 1120. First a static background image 

is drawn. This is typically a building floor plan or other newX««*cos e-ysin e 

representation of the physical environment. Next the pro- newY«**sin e+>> cos e 
gram draws each path, intersection, and HotSpot represented 

in model database 104. The program displays an image 55 where 0 is the difference angle. 

frame associated with the default initial node in main view As shown in FIG. 5c, sin 6 and cos 0 represent compo- 

window 1110. The program further displays available nents Vx and Vy, respectively, of the incoming vector, so 

choices of outgoing branches from the initial node. Once the there is no need to compute the actual difference angle, 

user selects a branch to follow, playback begins. The pro- Finally, stage 430 determines whether the current frame is 

gram highlights the selected path and the path's destination 60 the last frame of the branch. If the test is true, stage 435 

intersection on the map and begins real-time display of the determines whether the user has chosen a branch to proceed 

image frame sequence. As execution continues along a on. Otherwise the stream state is set to Pause in stage 445 

branch, the user may pause and execute any HotSpot and operation 400 terminates. If a branch choice is available, 

encountered along the path. As execution reaches the end of the image frame sequence that corresponds to the chosen 

the path, the program displays available branch choices for 65 branch is initialized in stage 440, as illustrated in FIG. 6 

the upcoming node. The user makes a selection to continue (operation 600). First, the location of the next image frame 

execution. sequence is retrieved from the new branch in stage 610. 
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Stage 615 then determines whether the next sequence is document's contents in the pop up window. For streaming 

stored in a different file from the current sequence, in which types of data (i.e., audio clips, video clips or image 

case the stream of the old file is closed in stage 620 and a sequences), the program creates the stream and intializes a 

stream for the new file is opened in stage 625. The program popup window to the starting frame specified by the 

then seeks to the first frame of the sequence, in stage 630. 5 HotSpot. User controls are then attached to the popup 

The path on the map that corresponds to the new branch is window and the stream state is set to Run. If the HotSpot is 

highlighted in stage 635. Finally, the program sets the stream an interactive game or exhibit a database model file is 

state to Run in stage 640 and operation 600 terminates. loaded ' bul n0 ma P view win dow is created, since the map 

The final step in operation 400 is to render the next image represents a conceptual flow, 

frame in main view window 1110 in stage 450. 10 u Pause: A * any P°!**™« playback, the user may cause 

During image frame processing, the user can manipulate tl^Tn £Z S * ^ cu ^ ent / ram 1 e ll b y 

various controls to control playback. Tfce present invention S^ a =^ h ^l^ 50 * mam vl f™ dow ™ 

■ t , . * u • * . iL Upon receiving this event, the program sets the stream state 

interleaves handling of asynchronous input events with thc * b preV enting the image display function 

image frame processing. The following sections describe the from displaying subsequent frames 

processing of possible input events. 35 Play: 0 nce playback is performed, the user may cause 

Choose Branch: Each node in the model has one or more image framc disp i ay to resum c from the paused state by 

branches leading away from the node. In two instances, selecting Play control 1140 in main view window 1110. 

when the tour reaches a node and when an input event causes Upon receiving this event, the program sets the stream state 

a jump to a node, the user must choose a branch to follow to Run, thereby enabling the image display function to 

away from the node. The program displays branch choices 20 display subsequent frames. 

available to the user as control icons in main view window Fast Forward: During image frame playback, the user 

1110. The user selects one of the icons, indicating the may cause the program to increase the speed of frame 

preferred choice (FIG. 12). This action triggers the Choose display by selecting Fast Forward control 1170 in main view 

Branch input event. The program processes the event by window 1110. Upon receiving this event, the program dis- 

storing the chosen branch. The choice is then retrieved and 25 plays the remainder of the current image frame sequence 

used by the image frame processing loop as described above. (i.e., the current path) at double the normal speed rate (some 

Jump to Node: At any point, the user may cause the image frames may be skipped during this operation), 

frame display to jump to an arbitrary node in the model by Fast Reverse: During image frame playback, the user may 

selecting an associated intersection in map view window cause the program to display image frames in reverse order 

1120. This action triggers a Jump to Node input event. Upon 30 by selecting Fast Reverse control 1160 in main view window 

receiving the event, the program identifies the image frame 1110. Upon receiving this event, the program displays the 

associated with the selected node. If the new image frame is current image frame sequence in reverse order at double the 

stored in a different file than the previous image frame normal speed rate (some frames may be skipped during this 

sequence, the program closes the previous input stream and operation). 

opens a stream to the new image file. The program then sets 35 Browse: At any point, the user may cause the program to 

the stream state to Pause and displays the image frame scan through all of the image frames in the model by use of 

associated with the new node in main view window 1110. the Browse control 1180 in main view window 1110. Browse 

Note that this image frame is the final frame of one of the control 1180 is continuously variable over a range of values, 

incoming branches. The program highlights the associated When Browse control 1180 is activated an event is opened 

path in map view window 1120 to help orient the user at the 40 that contains two values: a state value that indicates whether 

new location. The program then calculates branch choices the control is engaged or disengaged and an index value that 

for the node and displays them in a correct orientation indicates the position of the control. Upon receiving this 

relative to the user's view point (FIG. 11). event, the program sets the stream state to Pause. If the 

Jump to HotSpot: At any point, the user may cause the control state is Engaged, the program maps the index value 

image frame display to jump to an arbitrary HotSpot in the 45 to one of the branches in the model and highlights the 

model by selecting an associated HotSpot icon (e.g., associated path on the map. If the control state is 

HotSpot icons 1320 and 1520 in FIGS. 13 and 15, Disengaged, the program maps the index value and high- 

respectively) in map view window 1120. This action triggers lights a path as described above. The program further 

a Jump to HotSpot input event. The processing of this event calculates a particular image frame within the branch's 

is similar to the processing of a Jump to Node event. The 50 frame segment, retrieves that frame, and displays the frame 

program initializes the image frame sequence for the branch in main view window 1110. 

on which the HotSpot is located. The program then displays The image frame calculation is based on an arbitrary 

a particular image frame associated with the HotSpot and serial ordering of all branches in the model. This function 

highlights the new path on the map. Finally, the program maps the range of Browse index values over the resulting 

overlays the HotSpot's bounded region of interest on the 55 serial sequence. A specific index value is then mapped to an 

displayed video frame and highlights the HotSpot icon on offset within a specific branch, the image frame at that offset 

the map (FIGS. 13 and 15). in the branch's associated image frame sequence is then 

Open HotSpot: Whenever a HotSpot region of interest is identified. FIG. 7 illustrates this function. Paths 705, 706, 

displayed in main view window 1110, the user may open the and 707 represent the branches of a simple model. They are 

corresponding HotSpot by selecting the region of interest. 60 associated with frame sequences 702, 703, and 704, respec- 

This action triggers an Execute HotSpot event. Upon receiv- tively. A range of index values 701 maps to the three 

ing this event, the program first determines the type of sequences, as shown in FIG. 7. For example, an index value 

document (e.g. text, image, audio) referred by the HotSpot. of 60 falls within the subrange that maps to sequence 703, 

The program then pops up an appropriate type of window to and the program highlights path 706 accordingly. Once the 

display the data contained in the document (e.g., popup 65 offset of the given index value within the subrange is 

windows 1410 and 1610 in FIGS. 14 and 16, respectively). calculated the index value is further, it is straightforward to 

The program then opens the document and displays the mapping to frame 5 in frame sequence 703. 
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FIGS. 9-10 show the architecture of the application programming or geometric modeling by constructing a 

software classes that implement the present invention. As graph of decision nodes with branches. The outgoing 

illustrated in FIG. 9, the application software comprises a set branches of a node represent actions or decisions that are 

of classes derived from the Microsoft Foundation Classes taken in the flow of the game. Each outgoing branch has a 

(MFC), as well as application classes that make up a 5 distinct orientation vector. The game associates each unique 

separate application library. In addition to inheritance rela- vector with a user interface control. For example, assume a 

tionships between classes, the application specific classes simple control device with three buttons (e.g. a three-button 

are related to one another through pointer references. MFC- mouse). All vectors in the graph that are oriented at 90 

derived classes (VITAppObj, CMultiDocTemplate, degrees could be controlled by interface button A, all vectors 

CNet Player Doc, C Net Player View, CN odes View, 10 at 45 degrees by button B, and all vectors at 180 degrees by 

CChildFrame, and CPlayCtrls) provide the operations that button C. 

are specific to Microsoft Windows user interface. Examples Assume the game is initialized to start at some default 

of such operations are creating and deleting windows and node in the graph. If from that location the user pushes 

receiving events from the pointing device. button A, the game follows the branch oriented at 90 

Application specific library classes implement the core 35 degrees. This brings the user to the node at the end of this 

functionality of the program of the present invention. For branch. If at this node the user presses button B, the branch 

instance, these classes implement the playback capability. at 45 degrees is followed, thus connecting to the next node. 

The role of each of the application library classes is as If there is no branch that corresponds to the input, the game 

described below: position is not advanced. The game may also have discon- 

CDocHelper: This class creates and stores each of the 20 nected nodes (nodes that cannot be reached by a branch in 

stream objects (CMMStream) needed by the document. This the graph) or, in fact, multiple disjunct graphs. In this case, 

class also, keeps track of the currently active stream. the user interface must provide the capability to pass node 

CNodesNetDB: This class stores and provides access to identifiers to the program in order to effect jumps to disjunct 

the elements of the model. nodes. Note that in this embodiment, the graph is not 

CStreamViewHelper: This class performs processing 25 displayed as a map since it functions as a decision graph, 

required for stream views (the main view and streaming rather than as a map of a spatial environment, 

popup views). This class further interprets and processes It will be apparent to those skilled in the art that the same 

events (e.g. user controls and HotSpot invocation) received program code is used to traverse the graphs in both embodi- 

from one of these views by class CnetPlayerView. ments. The difference between the environments resides in 

CMMStream: This class performs processing required for 30 the format of the user interface, 

streams (e.g. opening and closing stream files). Furthermore, application can be extended to support 

CNodesViewHelper: This class performs processing multiple players. Players navigating the same model can be 

required for the map view. This class further interprets and represented on the map with different symbols and placed on 

processes events (e.g., jumping to intersections) received appropriate locations on the branches they travel on. 

from the map view by class CnodesView. 35 Application can be integrated with other collaborative 

CNodesNetGraph: This class draws model elements in programs such as chat (text or voice), (e.g., Netmeeting). 

map view window 1120. Players can then open a communication channel with other 

A set of application classes derived from the application players by simply clicking on symbols corresponding to the 

library classes add editing operations to the basic library other players. 

operations. For example, these classes implement the 40 Furthermore, computer-generated graphics can be inte- 
authoring capability. FIG. 10 illustrates the relationships grated with the video to allow for more complex animations 
between library classes and authoring classes. It will be composited within video to support interactive 3-D games, 
apparent to those skilled in the art that the operations of the The present invention can also be applied to training 
playback program are a subset of those of the authoring materials for complex procedures. For example, a video 
program. That is, the authoring program contains every 45 illustrating the actual procedure taking place can be shown 
operation of the playback program, plus additional opera- next to a conceptual flow of the presentation shown in map 
tions. These classes, and the additional operations that they view window 1120 with the branches highlighting the cur- 
provide, are described below: rent stage in the procedure. Additional types of hot spots can 

CDocHelperA: This class provides the capability to select be inserted by user such as personal notes and links to other 

a bitmap file (static background) for the document. 50 pertinent information to help the learning process. 

CNodesNetDBA: This class modifies the model database, In addition, video games or interactive exhibits can be 

both adding and deleting model element data. scripted by building the network of decision points and 

CStreamViewHelperA: This class provides operations to branches, with no special programming (already demon- 
modify stream views, specifically adding HotSpots to the strated in the current embodiment), 
view. This class invokes operations in CNodesNetDBA to 55 Furthermore, stories with multiple alternative endings can 
update the database for new HotSpots. be developed in accordance to the principles of the inven- 

CNodesViewHelperA: This class provides operations to tion. For instance, an application can be built where kids 

modify the map view, specifically adding, deleting, and assemble segments of the story in different ways to create 

editing the model elements that are represented in map view new stories. 

window 1120 (e.g., nodes, branches, branch links). This 60 Virtual shopping mall and virtual stores can also be 

class further invokes operations in CNodeNetGraphA to developed in accordance to the principles of the invention, 

update the drawings of model elements in the map view. Currently on-line shoppers must know in advance exactly 

CNodeNetGraphA: This class provides operations to what they want to search within a category. By associating 

modify the drawings of model elements in the map view. Hot spots to items shown in a moving video of a real store, 

A further embodiment of the present invention uses the 65 the computer system of the present invention replicates a 

model as the underlying representation of a video game. A real life shopping experience, allowing, for example, 

video interactive game can be scripted entirely without impulse buying. Users can obtain more information about an 
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item shown in the video by simply clicking on a correspond- 
ing hot spot. The item is, in turn, associated with unique 
URL of the site's e -commerce database. Users can thus add 
the item to their electronic shopping basket. 

Finally, the computer system of the present invention is 5 
also suitable for high-bandwidth applications. Video files 
encoded according to a broadcast quality standard such as 
MPEG2 can be streamed over high-speed networks to allow 
virtual visits to remote sites (e.g., museums, historic sites, 
etc). io 

Embodiments described above illustrate but do not limit 
the invention. In particular, the invention is not limited by 
any particular graphical user interface. In fact, any graphical 
user interface known in the art to perform the functions 
described herein can be used in accordance to the principles IS 
of the invention. In addition, the invention is not limited to 
any particular hardware/software implementation. Those 
skilled in the art realize that alternative hardware software 
implementations can be employed in lieu of the one 
described herein in accordance to the principles of the 20 
present invention. Other embodiments and variations are 
within the scope of the invention, as defined by the follow- 
ing claims. 

I claim: 

1. A computer system for navigating a virtual 25 
environment, the computer system comprising: 

a computer; 

a display device connected to the computer; 
one or more input devices connected to the computer; and 
a computer program executing on the computer, wherein 

the computer program comprises computer instructions 

for: 

generating a virtual environment from a real 
environment, the virtual environment comprising a 35 
plurality of nodes connected by branches, wherein 
each node represents an intersection in the real 
environment and each branch represents a path con- 
necting intersections in the real environment; 

displaying a map of at least a portion of the nodes and 4Q 
branches of the virtual environment in a first window 
of the display device; 

displaying a visual representation of the virtual envi- 
ronment in a second window of the display device; 
and 

enabling a user to navigate through the virtual envi- 
ronment by entering navigation commands via at 
least one input device. 

2. The computer system of claim 1, wherein each path in 
the map further comprises directional information, and the 5Q 
directional information is expressed relative to a current 
position in the map. 

3. The computer system of claim 1, wherein the computer 
program further comprises computer instructions for: 

associating multimedia content with a portion of the 55 
virtual environment; 

displaying a visual representation of the multimedia con- 
tent as an element of the virtual environment displayed 
in the second window of the display device; 

detecting that the user has selected the visual representa- 60 
tion of the multimedia content via one of the input 
devices; and 

displaying the multimedia content in a window of the 
display device. 

4. The computer system of claim 3, wherein the multi- 65 
media content is displayed in a third window of the display 
device. 



45 



5. The computer system of claim 3, wherein the multi- 
media content further comprises a web page. 

6. The computer system of claim 3, wherein the multi- 
media content further comprises a video clip. 

7. A computer system for composing a virtual 
environment, the computer system comprising: 

a computer; 

a display device connected to the computer; 

one or more input devices connected to the computer; and 

a computer program executing on the computer, wherein 

the computer program comprises computer instructions 

for: 

generating a virtual environment from a real 
environment, the virtual environment comprising a 
plurality of nodes connected by branches; 

enabling a user to associate an intersection of the real 
environment with a node of the virtual environment; 
and 

enabling a user to associate a visual representation of a 
path connecting intersections of the real environment 
with a branch of the virtual environment. 

8. The computer system of claim 7, wherein the computer 
program further comprises computer instructions for 
enabling a user to associate multimedia content with an 
element of the virtual environment. 

9. The computer system of claim 7, wherein the computer 
program further comprises computer instructions for editing 
associations between nodes and intersections and between 
branches and paths. 

10. A method for navigating a virtual environment gen- 
erated from a real environment, the virtual environment 
comprising a plurality of nodes connected by branches, 
wherein each node represents an intersection in the real 
environment and each branch represents a path connecting 
intersections in the real environment, the method compris- 
ing: 

displaying a map of at least a portion of the nodes and 
branches of the virtual environment in a first window of 
a display device; 

displaying a visual representation of the virtual environ- 
ment in a second window of the display device; and 

a user navigating through the virtual environment by 
entering navigation commands via at least one input 
device connected to the display device. 

11. The method of claim 10, wherein each path in the map 
further comprises directional information, and the direc- 
tional information is expressed relative to a current position 
in the map. 

12. The method of claim 10, further comprising: 
associating multimedia content with a portion of the 

virtual environment; 

displaying a visual representation of the multimedia con- 
tent as an element of the virtual environment displayed 
in the second window of the display device; 

the user selecting the visual representation of the multi- 
media content via one of the input devices; and 

displaying the multimedia content in a window of the 
display device. 

13. The method of claim 12, wherein the multimedia 
content is displayed in a third window of the display device. 

14. The method of claim 12, wherein the multimedia 
content further comprises a web page. 

15. The method of claim 12, wherein the multimedia 
content further comprises a video clip. 

16. A method for composing a virtual environment, the 
method comprising: 
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generating a virtual environment from a real environment, 

the virtual environment comprising a plurality of nodes 

connected by branches; 
a user associating an intersection of the real environment 

with a node of the virtual environment; and 5 
the user associating a visual representation of a path 

connecting intersections of the real environment with a 

branch of the virtual environment. 

17. The method of claim 16, further comprising the user 
associating multimedia content with an element of the 10 
virtual environment. 

18. The method of claim 16, further comprising editing 
associations between nodes and intersections and between 
branches and paths. 

19. A computer-readable storage medium comprising 15 
computer instructions for: 

generating a virtual environment from a real environment, 
the virtual environment comprising a plurality of nodes 
connected by branches, wherein each node represents 
an intersection in the real environment and each branch 20 
represents a path connecting intersections in the real 
environment; 

displaying a map of at least a portion of the nodes and 
branches of the virtual environment in a first window of 
a display device; 25 

displaying a visual representation of the virtual environ- 
ment in a second window of the display device; and 

enabling a user to navigate through the virtual environ- 
ment by entering navigation commands via at least one 
input device. 30 

20. The computer-readable storage medium of claim 19, 
wherein each path in the map further comprises directional 
information, and the directional information is expressed 
relative to a current position in the map. 

21. The computer-readable storage medium of claim 19, 35 
further comprising computer instructions for: 

associating multimedia content with a portion of the 
virtual environment; 

displaying a visual representation of the multimedia con- 
tent as an element of the virtual environment displayed 40 
in the second window of the display device; 

detecting that the user has selected the visual representa- 
tion of the multimedia content via one of the input 
devices; and 

45 

displaying the multimedia content in a window of the 
display device. 

22. The computer-readable storage medium of claim 21, 
wherein the multimedia content is displayed in a third 
window of the display device. 5Q 

23. The computer-readable storage medium of claim 21, 
wherein the multimedia content further comprises a web 
page. 

24. The computer-readable storage medium of claim 21, 
wherein the multimedia content further comprises a video 55 
clip. 

25. A computer-readable storage medium comprising 
computer instructions for: 

generating a virtual environment from a real environment, 

the virtual environment comprising a plurality of nodes 60 

connected by branches; 
enabling a user to associate an intersection of the real 

environment with a node of the virtual environment; 

and 

enabling the user to associate a visual representation of a 65 
path connecting intersections of the real environment 
with a branch of the virtual environment. 



26. The computer- readable storage medium of claim 25, 
further comprising computer instructions for enabling the 
user to associate multimedia content with an element of the 
virtual environment. 

27. The computer-readable storage medium of claim 25, 
further comprising computer instructions for enabling the 
user to edit associations between nodes and intersections and 
between branches and paths, 

28. A computer system for navigating a virtual 
environment, the computer system comprising: 

a computer; 

a display device connected to the computer; 

one or more input devices connected to the computer; and 
a computer program executing on the computer, 
wherein the computer program comprises computer 
instructions for: 

displaying a map of at least a portion of a virtual 
environment generated from a real environment in a 
first window of the display device; 

displaying a visual representation of the virtual envi- 
ronment in a second window of the display device; 
and 

highlighting a branch in the map displayed in the first 
window while a visual representation of a corre- 
sponding path in the real environment is displayed in 
the second window; 

detecting that a user has selected an intersection on the 
map; 

in response to the user selecting the intersection, dis- 
playing a corresponding visual representation of the 
intersection in the virtual environment in the second 
window; 

highlighting an icon displayed in the first window and 
corresponding to an element of the virtual environ- 
ment associated with multimedia data while the 
element is displayed in the second window; 

detecting that a user has selected a visual representation 
of an element of the virtual environment associated 
with multimedia data; and 

in response to the user selecting the element, displaying 
the multimedia data in a window of the display 
device. 

29. The computer system of claim 28, wherein the com- 
puter program is a video game. 

30. The computer system of claim 28, wherein the virtual 
environment further comprises a visual representation of an 
exhibit. 

31. A method of navigating a virtual environment gener- 
ated from a real environment, the method comprising: 

displaying a map of at least a portion of the virtual 
environment in a first window of a display device of a 
computer; 

displaying a visual representation of the virtual environ- 
ment in a second window of the display device; and 

highlighting a branch in the map displayed in the first 
window while a visual representation of a correspond- 
ing path in the real environment is displayed in the 
second window; 

detecting that a user has selected an intersection on the 
map; 

in response to the user selecting the intersection, display- 
ing a corresponding visual representation of the inter- 
section in the virtual environment in the second win- 
dow; 

highlighting an icon displayed in the first window and 
corresponding to an element of the virtual environment 
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associated with multimedia data while the element is 
displayed in the second window; 
detecting that a user has selected a visual representation of 
an element of the virtual environment associated with 
multimedia data; and 5 

in response to the user selecting the element, displaying 
the multimedia data in a window of the display device. 

32. A computer-readable storage medium comprising 
computer instructions for: 

displaying a map of at least a portion of a virtual envi- 
ronment generated from a real environment in a first 
window of a display device of a computer; 

displaying a visual representation of the virtual environ- 
ment in a second window of the display device; and ]5 

highlighting a branch in the map displayed in the first 
window while a visual representation of a correspond- 
ing path in the real environment is displayed in the 
second window; 

detecting that a user has selected an intersection on the 20 
map; 

in response to the user selecting the intersection, display- 
ing a corresponding visual representation of the inter- 
section in the virtual environment in the second win- 
dow; 

highlighting an icon displayed in the first window and 
corresponding to an element of the virtual environment 
associated with multimedia data while the element is 
displayed in the second window; 
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detecting that a user has selected a visual representation of 
an element of the virtual environment associated with 
multimedia data; and 
in response to the user selecting the element, displaying 
the multimedia data in a window of the display device. 
33. A computer system for navigating a vixtual 
environment, the computer system comprising a 

server computer connected to at least one client computer 

via a global-area network; 
wherein the client computer includes a display device, 
one or more input devices, and a computer program 
executable on the client computer, wherein the com- 
puter program comprises computer instructions for: 
retrieving a virtual environment generated from a real 
environment from a storage of the server computer, 
the virtual environment comprising a plurality of 
nodes connected by branches, wherein each node 
represents an intersection in the real environment 
and each branch represents a path connecting inter- 
sections in the real environment; 
displaying a map of at least a portion of the virtual 
environment in a first window of the display device; 
displaying a visual representation of the virtual envi- 
ronment in a second window of the display device; 
and 

enabling a user to navigate through the virtual envi- 
ronment by entering navigation commands via at 
least one input device. 
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It is certified that error appears in the above- identified patent and that said Letters Patent is 
hereby corrected as shown below: 



Drawings. 

Replace FIG. 13, with attached FIG 13. 
Replace FIG. 15, with attached FIG 15. 

Column 10. 

Line 64, replace sentence "Once the offset of the given index value within the subrange 
is calculated the index value is further, it is straightforward to mapping to frame 5 in 
frame sequence 703." with - Once the offset of the given index value within the sub- 
range is calculated, it is straightforward to map to frame 5 in frame sequence 703 --. 

Column 18, 

Line 33, replace word "vixtual" with -- virtual --. 
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JAMES E. ROGAN 
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